Papers with encoder model
INDUS: Effective and Efficient Language Models for Scientific Applications (2024.emnlp-industry)
Copied to clipboard
Bishwaranjan Bhattacharjee, Aashka Trivedi, Masayasu Muraoka, Muthukumaran Ramasubramanian, Takuma Udagawa, Iksha Gurung, Nishan Pantha, Rong Zhang, Bharath Dandala, Rahul Ramachandran, Manil Maskey, Kaylin Bugbee, Michael Little, Elizabeth Fancher, Irina Gerasimov, Armin Mehrabian, Lauren Sanders, Sylvain Costes, Sergi Blanco-Cuaresma, Kelly Lockhart, Thomas Allen, Felix Grezes, Megan Ansdell, Alberto Accomazzi, Yousef El-Kurdi, Davis Wertheimer, Birgit Pfitzmann, Cesar Berrospi Ramis, Michele Dolfi, Rafael Lima, Panagiotis Vagenas, S. Mukkavilli, Peter Staar, Sanaz Vahidinia, Ryan McGranaghan, Tsengdar Lee
| Challenge: | Large language models trained on general domain corpora showed remarkable results on natural language processing tasks. |
| Approach: | They develop a suite of large language models trained on general domain corpora that address NLP tasks and smaller versions of them created using knowledge distillation. |
| Outcome: | The proposed models outperform general-purpose and domain-specific encoders on new and existing tasks and in industrial settings. |
Seeing Beyond: Enhancing Visual Question Answering with Multi-Modal Retrieval (2025.coling-industry)
Copied to clipboard
| Challenge: | Multi-modal Large language models still suffer from model hallucination and lack of specific knowledge when answering challenging questions. |
| Approach: | They propose to use a multi-modal retrieval augmented generation method to integrate knowledge from all modalities into a model to enable alignment between query and knowledge. |
| Outcome: | The proposed method achieves significant performance improvement on the VQA dataset. |
BERT-Flow-VAE: A Weakly-supervised Model for Multi-Label Text Classification (2022.coling-1)
Copied to clipboard
| Challenge: | Multi-label Text Classification (MLTC) is a task of categorizing documents into one or more topics. Fully-supervised learning methods are undesirable for this task because of the diversity of domains of application and cost of manual labelling. |
| Approach: | They propose a Weakly-Supervised Multi-Label Text Classification model that produces BERT sentence embeddings and calibrates them using a flow model. |
| Outcome: | The proposed model outperforms baseline models in key metrics and achieves 84% performance on multi-label datasets. |
Manifold-Preserving Transformers are Effective for Short-Long Range Encoding (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Multi-head self-attention-based Transformers have shown promise in different learning tasks . but encoders of Transformers and their variants fail to preserve layer-wise contextual information . |
| Approach: | They propose an encoder model that guarantees a theoretical bound for layer-wise distance preservation between a pair of tokens. |
| Outcome: | The proposed model preserves equivalence between tokens and performs better than Transformers. |
One Agent To Rule Them All: Towards Multi-agent Conversational AI (2022.findings-acl)
Copied to clipboard
Christopher Clarke, Joseph Peper, Karthik Krishnamurthy, Walter Talamonti, Kevin Leach, Walter Lasecki, Yiping Kang, Lingjia Tang, Jason Mars
| Challenge: | Increasing volume of conversational agents (CAs) on the market has resulted in users being burdened with learning and adopting multiple agents to accomplish their tasks. |
| Approach: | They propose a task BBAI: Black-Box Agent Integration that integrates multiple black-box CAs at scale. |
| Outcome: | The proposed system outperforms existing benchmarks in the BBAI: Black-Box Agent Integration task. |
Scientific Paper Extractive Summarization Enhanced by Citation Graphs (2022.emnlp-main)
Copied to clipboard
| Challenge: | citation graphs can be used to extract scientific papers under different conditions. |
| Approach: | They propose a multi-granularity unsupervised summarization model that fine tunes a pre-trained encoder model on the citation graph by link prediction tasks. |
| Outcome: | The proposed model outperforms baseline models on a public benchmark dataset. |
SuperGLEBer: German Language Understanding Evaluation Benchmark (2024.naacl-long)
Copied to clipboard
| Challenge: | a new set of German-pretrained models are being released, but no established, diverse and systematic evaluation suite is available for them. |
| Approach: | They assemble a Natural Language Understanding benchmark suite for the German language and evaluate 10 existing German-pretrained models. |
| Outcome: | The proposed benchmark suite evaluates 10 German-pretrained models on 29 tasks . the results show that encoder models are good choices for most tasks, but not all . |
Enabling Natural Zero-Shot Prompting on Encoder Models via Statement-Tuning (2025.findings-naacl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) exhibit remarkable capabilities in zero-shot and few-shot settings, but they struggle with extending to few- shot and zero- shot settings due to their architectural design. |
| Approach: | They propose a technique that models discriminative tasks as a set of finite statements and trains an encoder model to discriminate between the potential statements to determine the label. |
| Outcome: | The proposed method achieves competitive performance compared to state-of-the-art LLMs with significantly fewer parameters. |
Enhancing Domain-Specific Encoder Models with LLM-Generated Data: How to Leverage Ontologies, and How to Do Without Them (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a new method for continual pretraining transformer encoder models is proposed for specialized domains with limited training data. |
| Approach: | They propose to use LLM-generated data to enrich domain-specific ontologies and pretrain transformer encoder models as an ontology-informed embedding model for concept definitions. |
| Outcome: | The proposed method improves on standard MLM pretraining on invasion biology domains. |
Searching by Code: A New SearchBySnippet Dataset and SnippeR Retrieval Model for Searching by Code Snippets (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing code search algorithms use code comments rather than full-text descriptions as text . existing algorithms use a code snippet and/or error traceback to find code . |
| Approach: | They propose a new search-by-code use case using a code snippet and error traceback . they propose implementing the search- by-code query in a StackOverflow dataset . |
| Outcome: | The proposed dataset outperforms strong baselines on SearchBySnippet with 0.451 Recall@10 . a code snippet and/or error traceback are used as queries to find bugs . |
Towards Fast and Accurate Modeling for Cross-Lingual Label Projection (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for label projection are inaccurate or slow for large-scale use. |
| Approach: | They propose to synthesize alignment sequence pairs and fine-tune an encoder model with span alignment objective while controlling data influence during training. |
| Outcome: | The proposed method outperforms state-of-the-art methods while maintaining fast inference speed across 50+ languages. |